{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S8VQGkQDR1MS"
      },
      "source": [
        "# Optimizing Vector Database Performance: Reducing Retrieval Latency with Quantization\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mongodb-developer/GenAI-Showcase/blob/main/notebooks/advanced_techniques/automatic_quantization_of_nomic_emebddings_with_mongodb.ipynb)\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xnhr8Dq9SGVI"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "This notebook explores techniques for optimizing vector database performance, focusing on reducing retrieval latency through the use of quantization methods. We examine the practical application of various embedding types\n",
        "- float32_embedding\n",
        "- int8_embedding\n",
        "- binary_embedding\n",
        "\n",
        "We analyze their impact on query precision and retrieval speed.\n",
        "\n",
        "By leveraging quantization strategies like scalar and binary quantization, we highlight the trade-offs between precision and efficiency.\n",
        "\n",
        "The notebook also includes a step-by-step demonstration of executing vector searches, measuring retrieval latencies, and visualizing results in a comparative framework.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hmsfJv-RTPvt"
      },
      "source": [
        "**Use Case:**\n",
        "\n",
        "The notebook demonstrates how to optimize vector database performance, specifically focusing on reducing retrieval latency using quantization methods.\n",
        "\n",
        "**Scenario**:\n",
        "You have a large dataset of text data (in this case, a book from Gutenberg) and you want to build a system that can efficiently find similar pieces of text based on a user's query.\n",
        "\n",
        "**Approach**:\n",
        "- Embeddings: The notebook uses SentenceTransformer to convert text into numerical vectors (embeddings) which capture the semantic meaning of the text.\n",
        "- Vector Database: MongoDB is used as a vector database to store and search these embeddings efficiently.\n",
        "- Quantization: To speed up retrieval, the notebook applies quantization techniques (scalar and binary) to the embeddings. This reduces the size of the embeddings, making searches faster but potentially impacting precision.\n",
        "Goal: By comparing the performance of different embedding types (float32, int8, binary), the notebook aims to show the trade-offs between retrieval speed and accuracy when using quantization. This helps in choosing the best approach for a given use case."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xx7exPnsSW0R"
      },
      "source": [
        "## Step 1: Install Libaries\n",
        "\n",
        "Here's a breakdown of the libraries and their roles:\n",
        "\n",
        "- **unstructured**: This library is used to process and structure various data formats, including text, enabling efficient analysis and extraction of information.\n",
        "- **pymongo**: This library provides the tools necessary to interact with MongoDB allowing for storage and retrieval of data within the project.\n",
        "- **nomic**: This library is used for vector embedding and other functions related to Nomic AI's models, specifically for generating and working with text embeddings.\n",
        "- **pandas**: This popular library is used for data manipulation and analysis, providing data structures and functions for efficient data handling and exploration.\n",
        "- **sentence_transformers**: This library is used for generating embeddings for text data using the SentenceTransformer model.\n",
        "\n",
        "By installing these packages, the code sets up the tools necessary for data processing, embedding generation, and storage with MongoDB.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HNRuam1cOC6j",
        "outputId": "f4ff39ea-06d5-4ff9-b001-1679e2da637d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: you may need to restart the kernel to use updated packages.\n"
          ]
        }
      ],
      "source": [
        "%pip install --quiet -U unstructured pymongo nomic pandas sentence_transformers einops"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "cAANfn5VPFiP"
      },
      "outputs": [],
      "source": [
        "import getpass\n",
        "import os\n",
        "\n",
        "\n",
        "# Function to securely get and set environment variables\n",
        "def set_env_securely(var_name, prompt):\n",
        "    value = getpass.getpass(prompt)\n",
        "    os.environ[var_name] = value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nho2rl6ESfUb"
      },
      "source": [
        "## Step 2: Data Loading and Preparation\n",
        "\n",
        "**Dataset Information**\n",
        "\n",
        "The dataset used in this example is \"Pushing to the Front,\" an ebook from Project Gutenberg. This book, focusing on self-improvement and success, is freely available for public use.\n",
        "\n",
        "The code leverages the ```unstructured``` library to process this raw text data, transforming it into a structured format suitable for semantic analysis and search. By chunking the text based on titles, the code creates meaningful units that can be embedded and stored in a vector database for efficient retrieval. This preprocessing is essential for building a robust and performant semantic search system.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V6YCStuykG_t"
      },
      "source": [
        "The code below ```requests``` library to fetch the text content of the book \"Pushing to the Front\" from Project Gutenberg's website. The URL points to the raw text file of the book."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "XvQPTTu8PIxP"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "\n",
        "url = \"https://www.gutenberg.org/cache/epub/21291/pg21291.txt\"\n",
        "response = requests.get(url)\n",
        "response.raise_for_status()\n",
        "book_text = response.text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xoLoA_ZnkMkv"
      },
      "source": [
        "Data Cleaning: The ```unstructured``` library is used to clean and structure the raw text. The ```group_broken_paragraphs``` function helps in combining fragmented paragraphs, ensuring better text flow.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "_vdvk7gLPKo4"
      },
      "outputs": [],
      "source": [
        "# import nltk\n",
        "# nltk.download('punkt')\n",
        "# nltk.download('averaged_perceptron_tagger')\n",
        "\n",
        "from unstructured.cleaners.core import group_broken_paragraphs\n",
        "from unstructured.partition.text import partition_text\n",
        "\n",
        "cleaned_text = group_broken_paragraphs(book_text)\n",
        "\n",
        "parsed_sections = partition_text(text=cleaned_text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pco6tIMDkZWs"
      },
      "source": [
        "The ```partition_text``` function further processes the cleaned text, dividing it into logical sections. These sections could represent chapters, sub-sections, or other meaningful units within the book."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JKK6DaLaPL9w",
        "outputId": "8db91645-dc85-4cb8-8190-37536cf383c4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The Project Gutenberg eBook of Pushing to the Front\n",
            "\n",
            "\n",
            "This ebook is for the use of anyone anywhere in the United States and most other parts of the world at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this ebook or online at www.gutenberg.org. If you are not located in the United States, you will have to check the laws of the country where you are located before using this eBook.\n",
            "\n",
            "\n",
            "Title: Pushing to the Front\n",
            "\n",
            "\n",
            "Author: Orison Swett Marden\n",
            "\n",
            "\n",
            "Release date: May 4, 2007 [eBook #21291]\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Show the first 5 sections\n",
        "for text in parsed_sections[:5]:\n",
        "    print(text)\n",
        "    print(\"\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6YI15B4l9pw"
      },
      "source": [
        "Chunking by Title: The ```chunk_by_title``` function identifies titles or headings within the parsed sections and uses them to create distinct chunks of text. This step is crucial for organizing the data into manageable units for subsequent embedding generation and semantic search.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "iX6mYTOPPNsG"
      },
      "outputs": [],
      "source": [
        "from unstructured.chunking.title import chunk_by_title\n",
        "\n",
        "chunks = chunk_by_title(parsed_sections)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GdglxgkFPPqf",
        "outputId": "c0d725cc-7456-4ebc-d0d6-46d45d50d0c3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The Project Gutenberg eBook of Pushing to the Front\n",
            "\n",
            "This ebook is for the use of anyone anywhere in the United States and most other parts of the world at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this ebook or online at www.gutenberg.org. If you are not located in the United States, you will have to check the laws of the country where you are located before using this eBook.\n"
          ]
        }
      ],
      "source": [
        "for chunk in chunks:\n",
        "    print(chunk)\n",
        "    break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kf-A1blUSj5n"
      },
      "source": [
        "## Step 3: Embeddings Generation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rW37dHf7PSF3",
        "outputId": "8d230f0a-9d41-4dc0-e17f-9cb20ebbdef2"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "<All keys matched successfully>\n"
          ]
        }
      ],
      "source": [
        "from sentence_transformers import SentenceTransformer\n",
        "from tqdm import tqdm\n",
        "\n",
        "embedding_model = SentenceTransformer(\n",
        "    \"nomic-ai/nomic-embed-text-v1.5\", trust_remote_code=True\n",
        ")\n",
        "\n",
        "# Determine the maximum sequence length for the model\n",
        "max_seq_length = embedding_model.max_seq_length\n",
        "\n",
        "\n",
        "def chunk_text(text, tokenizer, max_length=8192, overlap=50):\n",
        "    \"\"\"\n",
        "    Split the text into overlapping chunks based on token length.\n",
        "    \"\"\"\n",
        "    tokens = tokenizer.tokenize(text)\n",
        "    chunks = []\n",
        "    for i in range(0, len(tokens), max_length - overlap):\n",
        "        chunk_tokens = tokens[i : i + max_length]\n",
        "        chunk = tokenizer.convert_tokens_to_string(chunk_tokens)\n",
        "        chunks.append(chunk)\n",
        "    return chunks\n",
        "\n",
        "\n",
        "def get_embedding(text, task_prefix):\n",
        "    \"\"\"\n",
        "    Generate embeddings for a text string with a task-specific prefix.\n",
        "    \"\"\"\n",
        "\n",
        "    if not text.strip():\n",
        "        print(\"Attempted to get embedding for empty text.\")\n",
        "        return []\n",
        "\n",
        "    # Prepend the task instruction prefix to the text\n",
        "    prefixed_text = f\"{task_prefix}: {text}\"\n",
        "\n",
        "    # Get the tokenizer from the model\n",
        "    tokenizer = embedding_model.tokenizer\n",
        "\n",
        "    # Split text into chunks if it's too long\n",
        "    chunks = chunk_text(prefixed_text, tokenizer, max_length=max_seq_length)\n",
        "\n",
        "    # Embed each chunk\n",
        "    chunk_embeddings = embedding_model.encode(chunks)\n",
        "\n",
        "    # Return the first embedding as a list\n",
        "    return chunk_embeddings[0].tolist()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lZxbaszSiX_A"
      },
      "source": [
        "The embedding generation might take a approximately 20 minutes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "urSyjhUmPV1k",
        "outputId": "7c62a2ca-3c67-4c36-c6ab-e681929a19e6"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Generating embeddings: 100%|██████████| 4135/4135 [03:09<00:00, 21.77it/s]\n"
          ]
        }
      ],
      "source": [
        "from tqdm import tqdm\n",
        "\n",
        "# Pass chunks into embedding function with a progress bar\n",
        "embeddings = []\n",
        "# If you don't want to chunk the entire document simply slice the chunks\n",
        "# e.g for chunk in tqdm(chunks[:20], desc=\"Generating embeddings\")\n",
        "for chunk in tqdm(chunks, desc=\"Generating embeddings\"):\n",
        "    embedding = get_embedding(str(chunk), task_prefix=\"search_document\")\n",
        "    embeddings.append(embedding)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "82-hqKq9PXbj"
      },
      "outputs": [],
      "source": [
        "# Store the embedding data alongside the chunk, so a datapoing is {chunk:\"text\", embedding: \"embedding\"}\n",
        "embedding_data = []\n",
        "for chunk, embedding in zip(chunks, embeddings):\n",
        "    embedding_data.append(\n",
        "        {\n",
        "            \"chunk\": chunk.text,\n",
        "            \"float32_embedding\": embedding,\n",
        "            \"int8_embedding\": embedding,\n",
        "            \"binary_embedding\": embedding,\n",
        "        }\n",
        "    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "m4aWxhOQP5Nc"
      },
      "outputs": [],
      "source": [
        "# Convert the embedding data to a Pandas dataframe\n",
        "import pandas as pd\n",
        "\n",
        "dataset_df = pd.DataFrame(embedding_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FKWhXP-anM6z"
      },
      "source": [
        "When visualizing the dataset values, you will observe that the embedding attributes: float32_embedding, int_embedding and binary_emebedding all have the same values.\n",
        "\n",
        "In downstream proceses the values of the int_embedding and binary_embedding attributes for each data point will be modified to their respective data types, as a result of MongoDB Atlas auto quantization feature."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 293
        },
        "id": "RrFmAOJKP7U2",
        "outputId": "2d0b41e0-f3b3-4baa-8a4a-c609360e356a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunk</th>\n",
              "      <th>float32_embedding</th>\n",
              "      <th>int8_embedding</th>\n",
              "      <th>binary_embedding</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>﻿The Project Gutenberg eBook of Pushing to the...</td>\n",
              "      <td>[0.6708638668060303, 1.6244561672210693, -3.93...</td>\n",
              "      <td>[0.6708638668060303, 1.6244561672210693, -3.93...</td>\n",
              "      <td>[0.6708638668060303, 1.6244561672210693, -3.93...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Title: Pushing to the Front\\n\\nAuthor: Orison ...</td>\n",
              "      <td>[0.40157127380371094, 0.9250307679176331, -3.8...</td>\n",
              "      <td>[0.40157127380371094, 0.9250307679176331, -3.8...</td>\n",
              "      <td>[0.40157127380371094, 0.9250307679176331, -3.8...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>SAN JOSE\\n\\nCOPYRIGHT, 1911,\\n\\nBy ORISON SWET...</td>\n",
              "      <td>[0.8498777747154236, 1.207460880279541, -4.070...</td>\n",
              "      <td>[0.8498777747154236, 1.207460880279541, -4.070...</td>\n",
              "      <td>[0.8498777747154236, 1.207460880279541, -4.070...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>It has sent thousands of youths, with renewed ...</td>\n",
              "      <td>[0.21242937445640564, 0.9721437096595764, -3.2...</td>\n",
              "      <td>[0.21242937445640564, 0.9721437096595764, -3.2...</td>\n",
              "      <td>[0.21242937445640564, 0.9721437096595764, -3.2...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The author has received thousands of letters f...</td>\n",
              "      <td>[0.1889897882938385, 1.1587932109832764, -3.47...</td>\n",
              "      <td>[0.1889897882938385, 1.1587932109832764, -3.47...</td>\n",
              "      <td>[0.1889897882938385, 1.1587932109832764, -3.47...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                               chunk  \\\n",
              "0  ﻿The Project Gutenberg eBook of Pushing to the...   \n",
              "1  Title: Pushing to the Front\\n\\nAuthor: Orison ...   \n",
              "2  SAN JOSE\\n\\nCOPYRIGHT, 1911,\\n\\nBy ORISON SWET...   \n",
              "3  It has sent thousands of youths, with renewed ...   \n",
              "4  The author has received thousands of letters f...   \n",
              "\n",
              "                                   float32_embedding  \\\n",
              "0  [0.6708638668060303, 1.6244561672210693, -3.93...   \n",
              "1  [0.40157127380371094, 0.9250307679176331, -3.8...   \n",
              "2  [0.8498777747154236, 1.207460880279541, -4.070...   \n",
              "3  [0.21242937445640564, 0.9721437096595764, -3.2...   \n",
              "4  [0.1889897882938385, 1.1587932109832764, -3.47...   \n",
              "\n",
              "                                      int8_embedding  \\\n",
              "0  [0.6708638668060303, 1.6244561672210693, -3.93...   \n",
              "1  [0.40157127380371094, 0.9250307679176331, -3.8...   \n",
              "2  [0.8498777747154236, 1.207460880279541, -4.070...   \n",
              "3  [0.21242937445640564, 0.9721437096595764, -3.2...   \n",
              "4  [0.1889897882938385, 1.1587932109832764, -3.47...   \n",
              "\n",
              "                                    binary_embedding  \n",
              "0  [0.6708638668060303, 1.6244561672210693, -3.93...  \n",
              "1  [0.40157127380371094, 0.9250307679176331, -3.8...  \n",
              "2  [0.8498777747154236, 1.207460880279541, -4.070...  \n",
              "3  [0.21242937445640564, 0.9721437096595764, -3.2...  \n",
              "4  [0.1889897882938385, 1.1587932109832764, -3.47...  "
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MxD-hkXiqPqL"
      },
      "source": [
        "## Step 4: MongoDB (Operational and Vector Database)\n",
        "\n",
        "MongoDB acts as both an operational and vector database for the RAG system.\n",
        "MongoDB Atlas specifically provides a database solution that efficiently stores, queries and retrieves vector embeddings.\n",
        "\n",
        "Creating a database and collection within MongoDB is made simple with MongoDB Atlas.\n",
        "\n",
        "1. First, register for a [MongoDB Atlas account](https://www.mongodb.com/cloud/atlas/register). For existing users, sign into MongoDB Atlas.\n",
        "2. [Follow the instructions](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster/). Select Atlas UI as the procedure to deploy your first cluster.\n",
        "\n",
        "Follow MongoDB’s [steps to get the connection](https://www.mongodb.com/docs/manual/reference/connection-string/) string from the Atlas UI. After setting up the database and obtaining the Atlas cluster connection URI, securely store the URI within your development environment.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "32dtWQ4gqQNe",
        "outputId": "bd8b6c51-ee35-4761-c813-370bfabc098d"
      },
      "outputs": [],
      "source": [
        "# Set MongoDB URI\n",
        "set_env_securely(\"MONGO_URI\", \"Enter your MONGO URI: \")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "VvTIs5fqqTto"
      },
      "outputs": [],
      "source": [
        "import pymongo\n",
        "\n",
        "\n",
        "def get_mongo_client(mongo_uri):\n",
        "    \"\"\"Establish and validate connection to the MongoDB.\"\"\"\n",
        "\n",
        "    client = pymongo.MongoClient(\n",
        "        mongo_uri, appname=\"devrel.showcase.quantized_embeddings_nomic.python\"\n",
        "    )\n",
        "\n",
        "    # Validate the connection\n",
        "    ping_result = client.admin.command(\"ping\")\n",
        "    if ping_result.get(\"ok\") == 1.0:\n",
        "        # Connection successful\n",
        "        print(\"Connection to MongoDB successful\")\n",
        "        return client\n",
        "    else:\n",
        "        print(\"Connection to MongoDB failed\")\n",
        "    return None\n",
        "\n",
        "\n",
        "MONGO_URI = os.environ[\"MONGO_URI\"]\n",
        "if not MONGO_URI:\n",
        "    print(\"MONGO_URI not set in environment variables\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SCw0DOdpqVDP",
        "outputId": "b992c23b-4b25-405c-cbf3-5a125a4e0d2c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Connection to MongoDB successful\n",
            "Collection 'pushing_to_the_front_orison_quantized' created successfully.\n"
          ]
        }
      ],
      "source": [
        "from pymongo.errors import CollectionInvalid\n",
        "\n",
        "mongo_client = get_mongo_client(MONGO_URI)\n",
        "\n",
        "DB_NAME = \"career_coach\"\n",
        "COLLECTION_NAME = \"pushing_to_the_front_orison_quantized\"\n",
        "\n",
        "# Create or get the database\n",
        "db = mongo_client[DB_NAME]\n",
        "\n",
        "# Check if the collection exists\n",
        "if COLLECTION_NAME not in db.list_collection_names():\n",
        "    try:\n",
        "        # Create the collection\n",
        "        db.create_collection(COLLECTION_NAME)\n",
        "        print(f\"Collection '{COLLECTION_NAME}' created successfully.\")\n",
        "    except CollectionInvalid as e:\n",
        "        print(f\"Error creating collection: {e}\")\n",
        "else:\n",
        "    print(f\"Collection '{COLLECTION_NAME}' already exists.\")\n",
        "\n",
        "# Assign the collection\n",
        "collection = db[COLLECTION_NAME]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mJfggsulS0Lj"
      },
      "source": [
        "## Step 5: Data Ingestion"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pXKUZL8NRJvn",
        "outputId": "455fffb6-e480-4f75-db53-ea2bdb5e8f95"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DeleteResult({'n': 0, 'electionId': ObjectId('7fffffff0000000000000039'), 'opTime': {'ts': Timestamp(1734581901, 1), 't': 57}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1734581901, 1), 'signature': {'hash': b'\\xd8+\\x12\\xed3V\\x1bi\\x9df-\\xf8\\x06\\x97\\xd3p\\xa64\\x05A', 'keyId': 7390008424139849730}}, 'operationTime': Timestamp(1734581901, 1)}, acknowledged=True)"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "collection.delete_many({})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qeRnaMjyqngL",
        "outputId": "f5e0cfb5-f9b2-48ca-e37c-2249d2ec2d12"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Data ingestion into MongoDB completed\n"
          ]
        }
      ],
      "source": [
        "documents = dataset_df.to_dict(\"records\")\n",
        "collection.insert_many(documents)\n",
        "\n",
        "print(\"Data ingestion into MongoDB completed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nngpeSw2StLP"
      },
      "source": [
        "## Step 6: Vector Search Index Creation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "iigtZEtKqo9F"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "from pymongo.operations import SearchIndexModel\n",
        "\n",
        "\n",
        "def setup_vector_search_index(collection, index_definition, index_name=\"vector_index\"):\n",
        "    \"\"\"\n",
        "    Setup a vector search index for a MongoDB collection and wait for 30 seconds.\n",
        "\n",
        "    Args:\n",
        "    collection: MongoDB collection object\n",
        "    index_definition: Dictionary containing the index definition\n",
        "    index_name: Name of the index (default: \"vector_index\")\n",
        "    \"\"\"\n",
        "    new_vector_search_index_model = SearchIndexModel(\n",
        "        definition=index_definition, name=index_name, type=\"vectorSearch\"\n",
        "    )\n",
        "\n",
        "    # Create the new index\n",
        "    try:\n",
        "        result = collection.create_search_index(model=new_vector_search_index_model)\n",
        "        print(f\"Creating index '{index_name}'...\")\n",
        "\n",
        "        # Sleep for 60 seconds\n",
        "        print(f\"Waiting for 60 seconds to allow index '{index_name}' to be created...\")\n",
        "        time.sleep(60)\n",
        "\n",
        "        print(f\"60-second wait completed for index '{index_name}'.\")\n",
        "        return result\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error creating new vector search index '{index_name}': {e!s}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "MfciGsoKqz1H"
      },
      "outputs": [],
      "source": [
        "def create_vector_index_definition():\n",
        "    \"\"\"\n",
        "    Create a vector index definition with predefined quantization methods.\n",
        "\n",
        "    This function defines vector index fields with specific paths, dimensionalities,\n",
        "    and similarity metrics. It includes support for quantization methods:\n",
        "    - \"scalar\" quantization is applied to the \"int8_embedding\" field.\n",
        "    - \"binary\" quantization is applied to the \"binary_embedding\" field.\n",
        "    - No quantization is applied to the \"float32_embedding\" field.\n",
        "\n",
        "    Returns:\n",
        "      dict: A dictionary containing the vector index definition, including\n",
        "      fields with their respective paths, quantization methods, dimensions,\n",
        "      and similarity measures.\n",
        "    \"\"\"\n",
        "\n",
        "    # Define the field types\n",
        "    base_fields = [\n",
        "        {\n",
        "            \"type\": \"vector\",\n",
        "            \"path\": \"float32_embedding\",\n",
        "            \"numDimensions\": 768,\n",
        "            \"similarity\": \"cosine\",\n",
        "        },\n",
        "        {\n",
        "            \"type\": \"vector\",\n",
        "            \"path\": \"int8_embedding\",\n",
        "            \"quantization\": \"scalar\",\n",
        "            \"numDimensions\": 768,\n",
        "            \"similarity\": \"cosine\",\n",
        "        },\n",
        "        {\n",
        "            \"type\": \"vector\",\n",
        "            \"path\": \"binary_embedding\",\n",
        "            \"quantization\": \"binary\",\n",
        "            \"numDimensions\": 768,\n",
        "            \"similarity\": \"euclidean\",\n",
        "        },\n",
        "    ]\n",
        "\n",
        "    return {\"fields\": base_fields}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "1lnvm7jzq3QN"
      },
      "outputs": [],
      "source": [
        "vector_index_definition = create_vector_index_definition()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "POSDFMQrPfzD",
        "outputId": "6426c270-f928-4693-cd79-4203595cc4be"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'fields': [{'type': 'vector', 'path': 'float32_embedding', 'numDimensions': 768, 'similarity': 'cosine'}, {'type': 'vector', 'path': 'int8_embedding', 'quantization': 'scalar', 'numDimensions': 768, 'similarity': 'cosine'}, {'type': 'vector', 'path': 'binary_embedding', 'quantization': 'binary', 'numDimensions': 768, 'similarity': 'euclidean'}]}\n"
          ]
        }
      ],
      "source": [
        "print(vector_index_definition)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "_HCZvktJq8HN",
        "outputId": "a9aec482-8384-434b-b5b9-e8f1244dfd5b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating index 'vector_index'...\n",
            "Waiting for 60 seconds to allow index 'vector_index' to be created...\n",
            "60-second wait completed for index 'vector_index'.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'vector_index'"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "setup_vector_search_index(collection, vector_index_definition, \"vector_index\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIz7BRFIS5LF"
      },
      "source": [
        "## Step 7: Vector Search Operation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "metadata": {
        "id": "vJj41wyfOegY"
      },
      "outputs": [],
      "source": [
        "def custom_vector_search(\n",
        "    user_query, collection, embedding_path, vector_search_index_name=\"vector_index\"\n",
        "):\n",
        "    \"\"\"\n",
        "    Perform a vector search in the MongoDB collection based on the user query.\n",
        "\n",
        "    Args:\n",
        "        user_query (str): The user's query string.\n",
        "        collection (MongoCollection): The MongoDB collection to search.\n",
        "        embedding_path (str): The path of the embedding field in the documents.\n",
        "        vector_search_index_name (str): The name of the vector search index.\n",
        "\n",
        "    Returns:\n",
        "        list: A list of matching documents.\n",
        "    \"\"\"\n",
        "\n",
        "    # Generate embedding for the user query\n",
        "    query_embedding = get_embedding(user_query, task_prefix=\"search_query\")\n",
        "\n",
        "    if query_embedding is None:\n",
        "        return \"Invalid query or embedding generation failed.\"\n",
        "\n",
        "    # Define the vector search stage\n",
        "    vector_search_stage = {\n",
        "        \"$vectorSearch\": {\n",
        "            \"index\": vector_search_index_name,  # Specifies the index to use for the search\n",
        "            \"queryVector\": query_embedding,  # The vector representing the query\n",
        "            \"path\": embedding_path,  # Field in the documents containing the vectors to search against\n",
        "            \"numCandidates\": 1000,  # Number of candidate matches to consider\n",
        "            \"limit\": 10,  # Return top 5 matches\n",
        "        }\n",
        "    }\n",
        "\n",
        "    project_stage = {\n",
        "        \"$project\": {\n",
        "            \"_id\": 0,  # Exclude the _id field\n",
        "            \"chunk\": 1,\n",
        "            \"score\": {\n",
        "                \"$meta\": \"vectorSearchScore\"  # Include the search score\n",
        "            },\n",
        "        }\n",
        "    }\n",
        "\n",
        "    # Define the aggregate pipeline with the vector search stage and additional stages\n",
        "    pipeline = [vector_search_stage, project_stage]\n",
        "\n",
        "    # Execute the explain command\n",
        "    explain_result = collection.database.command(\n",
        "        \"explain\",\n",
        "        {\"aggregate\": collection.name, \"pipeline\": pipeline, \"cursor\": {}},\n",
        "        verbosity=\"executionStats\",\n",
        "    )\n",
        "\n",
        "    # Extract the execution time\n",
        "    vector_search_explain = explain_result[\"stages\"][0][\"$vectorSearch\"]\n",
        "    execution_time_ms = vector_search_explain[\"explain\"][\"query\"][\"stats\"][\"context\"][\n",
        "        \"millisElapsed\"\n",
        "    ]\n",
        "\n",
        "    # Execute the actual query\n",
        "    results = list(collection.aggregate(pipeline))\n",
        "\n",
        "    return results, execution_time_ms"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "metadata": {
        "id": "ZdPmgxjKP6E3"
      },
      "outputs": [],
      "source": [
        "def run_vector_search_operations(\n",
        "    user_query, collection, vector_search_index_name=\"vector_index\"\n",
        "):\n",
        "    \"\"\"\n",
        "    Run vector search operations for different embedding paths and store results in a DataFrame.\n",
        "    \"\"\"\n",
        "    embedding_paths = [\"float32_embedding\", \"int8_embedding\", \"binary_embedding\"]\n",
        "    results_data = []\n",
        "\n",
        "    for path in embedding_paths:\n",
        "        try:\n",
        "            results, execution_time_ms = custom_vector_search(\n",
        "                user_query=user_query,\n",
        "                collection=collection,\n",
        "                embedding_path=path,\n",
        "                vector_search_index_name=vector_search_index_name,\n",
        "            )\n",
        "\n",
        "            # Format all results for this precision type into a single string\n",
        "            formatted_results = \"\\n\".join(\n",
        "                [f\"[{result['score']:.4f}] {result['chunk']}\" for result in results]\n",
        "            )\n",
        "\n",
        "            results_data.append(\n",
        "                {\n",
        "                    \"Precision (Data Type)\": path.split(\"_\")[0],\n",
        "                    \"Retrieval Latency (ms)\": f\"{execution_time_ms:.6f}\",\n",
        "                    \"Results\": formatted_results,\n",
        "                }\n",
        "            )\n",
        "\n",
        "        except Exception as e:\n",
        "            results_data.append(\n",
        "                {\n",
        "                    \"Precision (Data Type)\": path.split(\"_\")[0],\n",
        "                    \"Retrieval Latency (ms)\": \"Error\",\n",
        "                    \"Results\": str(e),\n",
        "                }\n",
        "            )\n",
        "\n",
        "    results_df = pd.DataFrame(results_data)\n",
        "\n",
        "    # Set display options for better visibility\n",
        "    pd.set_option(\"display.max_colwidth\", None)\n",
        "    pd.set_option(\"display.max_rows\", None)\n",
        "    pd.set_option(\"display.width\", None)\n",
        "\n",
        "    return results_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7PPDt-iTBIO"
      },
      "source": [
        "## Step 8: Retrieving Documents and Analysing Results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 82,
      "metadata": {
        "id": "q-LnwhRyQkKH"
      },
      "outputs": [],
      "source": [
        "# Perform the vector search and visualize the results\n",
        "user_query = \"How do I increase my productivity for maximum output\"\n",
        "results_df = run_vector_search_operations(user_query, collection)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "One key point to note: If you’ve followed this example with a small dataset, you likely won’t observe significant retrieval latency improvements. Quantization methods truly demonstrate their benefits when dealing with large-scale datasets—on the order of one million or more embeddings—where memory savings and speed gains become substantially more noticeable.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 83,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "K6MiUemIQppq",
        "outputId": "02f0a2e3-7146-4380-ebd5-062899759719"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Precision (Data Type)</th>\n",
              "      <th>Retrieval Latency (ms)</th>\n",
              "      <th>Results</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>float32</td>\n",
              "      <td>4.510477</td>\n",
              "      <td>[0.8310] Horace Greeley said that the best product of labor is the high-minded workman with an enthusiasm for his work.\\n\\n\"The best method is obtained by earnestness,\" said Salvini. \"If you can impress people with the conviction that you feel what you say, they will pardon many shortcomings. And above all, study, study, study! All the genius in the world will not help you along with any art, unless you become a hard student. It has taken me years to master a single part.\"\\n[0.8151] Adopt this motto as yours. Hang it up in your bedroom, in your office or place of business, put it into your pocket-book, weave it into the texture of everything you do, and your life-work will be what every one's should be--A MASTERPIECE.\\n\\nCHAPTER XXIII\\n\\nTHE REWARD OF PERSISTENCE\\n\\nEvery noble work is at first impossible.--CARLYLE.\\n[0.8134] \"Many persons seeing me so much engaged in active life,\" said Edward Bulwer Lytton, \"and as much about the world as if I had never been a student, have said to me, 'When do you get time to write all your books? How on earth do you contrive to do so much work?' I shall surprise you by the answer I made. The answer is this--'I contrive to do so much by never doing too much at a time. A man to get through work well must not overwork himself; or, if he do too much to-day, the reaction of fatigue\\n[0.8131] The way to get the best out of yourself is to put things right up to yourself, handle yourself without gloves, and talk to yourself as you would to a son of yours who has great ability but who is not using half of it.\\n\\nWhen you go into an undertaking just say to yourself, \"Now, this thing is right up to me. I've got to make good, to show the man in me or the coward. There is no backing out.\"\\n[0.8123] \"You are capable of something much better than what you are doing. You must start out to-day with a firm resolution to make the returns from your work greater to-night than ever before. You must make this a red-letter day. Bestir yourself; get the cobwebs out of your head; brush off the brain ash. Think, think, think to some purpose! Do not mull and mope like this. You are only half-alive, man; get a move on you!\"\\n[0.8123] Edwin Chadwick, in his report to the British Parliament, stated that children, working on half time (that is, studying three hours a day and working the rest of their time out of doors), really made the greatest intellectual progress during the year. Business men have often accomplished wonders during the busiest lives by simply devoting one, two, three, or four hours daily to study or other literary work.\\n[0.8121] Refresh, renew, rejuvenate yourself by play and pleasant recreation. Play as hard as you work; have a jolly good time, and then you will get that refreshing, invigorating sleep which gives an overplus of energy, a buoyancy of spirit which will make you eager to plunge into the next day's work.\\n[0.8110] A successful manufacturer says: \"If you make a good pin, you will earn more money than if you make a bad steam engine.\" \"If a man can write a better book, preach a better sermon, or make a better mousetrap than his neighbor,\" says Emerson, \"though he build his house in the woods, the world will make a path to his door.\"\\n[0.8103] No matter where you go, study the situation. Think why the man does not do better if he is not doing well, why he remains in mediocrity all his life. If he is making a remarkable success, try to find out why. Keep your eyes open, your ears open. Make deductions from what you see and hear. Trace difficulties; look up evidences of success or failure everywhere. It will be one of the greatest factors in your own success.\\n\\nCHAPTER XXX\\n\\nSELF\\n[0.8088] There is a sense of great power in a vocation after a man has reached the point of efficiency in it, the point of productiveness, the point where his skill begins to tell and brings in returns. Up to this point of efficiency, while he is learning his trade, the time seems to have been almost thrown away. But he has been storing up a vast reserve of knowledge of detail, laying foundations, forming his acquaintances, gaining his reputation for truthfulness, trustworthiness, and integrity, and in</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>int8</td>\n",
              "      <td>3.791106</td>\n",
              "      <td>[0.8307] Horace Greeley said that the best product of labor is the high-minded workman with an enthusiasm for his work.\\n\\n\"The best method is obtained by earnestness,\" said Salvini. \"If you can impress people with the conviction that you feel what you say, they will pardon many shortcomings. And above all, study, study, study! All the genius in the world will not help you along with any art, unless you become a hard student. It has taken me years to master a single part.\"\\n[0.8152] Adopt this motto as yours. Hang it up in your bedroom, in your office or place of business, put it into your pocket-book, weave it into the texture of everything you do, and your life-work will be what every one's should be--A MASTERPIECE.\\n\\nCHAPTER XXIII\\n\\nTHE REWARD OF PERSISTENCE\\n\\nEvery noble work is at first impossible.--CARLYLE.\\n[0.8135] \"Many persons seeing me so much engaged in active life,\" said Edward Bulwer Lytton, \"and as much about the world as if I had never been a student, have said to me, 'When do you get time to write all your books? How on earth do you contrive to do so much work?' I shall surprise you by the answer I made. The answer is this--'I contrive to do so much by never doing too much at a time. A man to get through work well must not overwork himself; or, if he do too much to-day, the reaction of fatigue\\n[0.8127] \"You are capable of something much better than what you are doing. You must start out to-day with a firm resolution to make the returns from your work greater to-night than ever before. You must make this a red-letter day. Bestir yourself; get the cobwebs out of your head; brush off the brain ash. Think, think, think to some purpose! Do not mull and mope like this. You are only half-alive, man; get a move on you!\"\\n[0.8126] The way to get the best out of yourself is to put things right up to yourself, handle yourself without gloves, and talk to yourself as you would to a son of yours who has great ability but who is not using half of it.\\n\\nWhen you go into an undertaking just say to yourself, \"Now, this thing is right up to me. I've got to make good, to show the man in me or the coward. There is no backing out.\"\\n[0.8125] Edwin Chadwick, in his report to the British Parliament, stated that children, working on half time (that is, studying three hours a day and working the rest of their time out of doors), really made the greatest intellectual progress during the year. Business men have often accomplished wonders during the busiest lives by simply devoting one, two, three, or four hours daily to study or other literary work.\\n[0.8122] Refresh, renew, rejuvenate yourself by play and pleasant recreation. Play as hard as you work; have a jolly good time, and then you will get that refreshing, invigorating sleep which gives an overplus of energy, a buoyancy of spirit which will make you eager to plunge into the next day's work.\\n[0.8105] No matter where you go, study the situation. Think why the man does not do better if he is not doing well, why he remains in mediocrity all his life. If he is making a remarkable success, try to find out why. Keep your eyes open, your ears open. Make deductions from what you see and hear. Trace difficulties; look up evidences of success or failure everywhere. It will be one of the greatest factors in your own success.\\n\\nCHAPTER XXX\\n\\nSELF\\n[0.8103] A successful manufacturer says: \"If you make a good pin, you will earn more money than if you make a bad steam engine.\" \"If a man can write a better book, preach a better sermon, or make a better mousetrap than his neighbor,\" says Emerson, \"though he build his house in the woods, the world will make a path to his door.\"\\n[0.8082] There is a sense of great power in a vocation after a man has reached the point of efficiency in it, the point of productiveness, the point where his skill begins to tell and brings in returns. Up to this point of efficiency, while he is learning his trade, the time seems to have been almost thrown away. But he has been storing up a vast reserve of knowledge of detail, laying foundations, forming his acquaintances, gaining his reputation for truthfulness, trustworthiness, and integrity, and in</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>binary</td>\n",
              "      <td>6.250622</td>\n",
              "      <td>[0.0032] Horace Greeley said that the best product of labor is the high-minded workman with an enthusiasm for his work.\\n\\n\"The best method is obtained by earnestness,\" said Salvini. \"If you can impress people with the conviction that you feel what you say, they will pardon many shortcomings. And above all, study, study, study! All the genius in the world will not help you along with any art, unless you become a hard student. It has taken me years to master a single part.\"\\n[0.0031] \"You are capable of something much better than what you are doing. You must start out to-day with a firm resolution to make the returns from your work greater to-night than ever before. You must make this a red-letter day. Bestir yourself; get the cobwebs out of your head; brush off the brain ash. Think, think, think to some purpose! Do not mull and mope like this. You are only half-alive, man; get a move on you!\"\\n[0.0030] Adopt this motto as yours. Hang it up in your bedroom, in your office or place of business, put it into your pocket-book, weave it into the texture of everything you do, and your life-work will be what every one's should be--A MASTERPIECE.\\n\\nCHAPTER XXIII\\n\\nTHE REWARD OF PERSISTENCE\\n\\nEvery noble work is at first impossible.--CARLYLE.\\n[0.0030] \"Many persons seeing me so much engaged in active life,\" said Edward Bulwer Lytton, \"and as much about the world as if I had never been a student, have said to me, 'When do you get time to write all your books? How on earth do you contrive to do so much work?' I shall surprise you by the answer I made. The answer is this--'I contrive to do so much by never doing too much at a time. A man to get through work well must not overwork himself; or, if he do too much to-day, the reaction of fatigue\\n[0.0030] No matter where you go, study the situation. Think why the man does not do better if he is not doing well, why he remains in mediocrity all his life. If he is making a remarkable success, try to find out why. Keep your eyes open, your ears open. Make deductions from what you see and hear. Trace difficulties; look up evidences of success or failure everywhere. It will be one of the greatest factors in your own success.\\n\\nCHAPTER XXX\\n\\nSELF\\n[0.0030] Edwin Chadwick, in his report to the British Parliament, stated that children, working on half time (that is, studying three hours a day and working the rest of their time out of doors), really made the greatest intellectual progress during the year. Business men have often accomplished wonders during the busiest lives by simply devoting one, two, three, or four hours daily to study or other literary work.\\n[0.0030] The way to get the best out of yourself is to put things right up to yourself, handle yourself without gloves, and talk to yourself as you would to a son of yours who has great ability but who is not using half of it.\\n\\nWhen you go into an undertaking just say to yourself, \"Now, this thing is right up to me. I've got to make good, to show the man in me or the coward. There is no backing out.\"\\n[0.0030] \"Let us not be content to mine the most coal, to make the largest locomotives, to weave the largest quantities of carpets; but, amid the sounds of the pick, the blows of the hammer, the rattle of the looms, and the roar of the machinery, take care that the immortal mechanism of God's own hand--the mind--is still full-trained for the highest and noblest service.\"\\n[0.0029] This striving for excellence will make you grow. It will call out your resources, call out the best thing in you. The constant stretching of the mind over problems which interest you, which are to mean everything to you in the future, will help you expand into a broader, larger, more effective man.\\n[0.0029] A successful manufacturer says: \"If you make a good pin, you will earn more money than if you make a bad steam engine.\" \"If a man can write a better book, preach a better sermon, or make a better mousetrap than his neighbor,\" says Emerson, \"though he build his house in the woods, the world will make a path to his door.\"</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  Precision (Data Type) Retrieval Latency (ms)  \\\n",
              "0               float32               4.510477   \n",
              "1                  int8               3.791106   \n",
              "2                binary               6.250622   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Results  \n",
              "0  [0.8310] Horace Greeley said that the best product of labor is the high-minded workman with an enthusiasm for his work.\\n\\n\"The best method is obtained by earnestness,\" said Salvini. \"If you can impress people with the conviction that you feel what you say, they will pardon many shortcomings. And above all, study, study, study! All the genius in the world will not help you along with any art, unless you become a hard student. It has taken me years to master a single part.\"\\n[0.8151] Adopt this motto as yours. Hang it up in your bedroom, in your office or place of business, put it into your pocket-book, weave it into the texture of everything you do, and your life-work will be what every one's should be--A MASTERPIECE.\\n\\nCHAPTER XXIII\\n\\nTHE REWARD OF PERSISTENCE\\n\\nEvery noble work is at first impossible.--CARLYLE.\\n[0.8134] \"Many persons seeing me so much engaged in active life,\" said Edward Bulwer Lytton, \"and as much about the world as if I had never been a student, have said to me, 'When do you get time to write all your books? How on earth do you contrive to do so much work?' I shall surprise you by the answer I made. The answer is this--'I contrive to do so much by never doing too much at a time. A man to get through work well must not overwork himself; or, if he do too much to-day, the reaction of fatigue\\n[0.8131] The way to get the best out of yourself is to put things right up to yourself, handle yourself without gloves, and talk to yourself as you would to a son of yours who has great ability but who is not using half of it.\\n\\nWhen you go into an undertaking just say to yourself, \"Now, this thing is right up to me. I've got to make good, to show the man in me or the coward. There is no backing out.\"\\n[0.8123] \"You are capable of something much better than what you are doing. You must start out to-day with a firm resolution to make the returns from your work greater to-night than ever before. You must make this a red-letter day. Bestir yourself; get the cobwebs out of your head; brush off the brain ash. Think, think, think to some purpose! Do not mull and mope like this. You are only half-alive, man; get a move on you!\"\\n[0.8123] Edwin Chadwick, in his report to the British Parliament, stated that children, working on half time (that is, studying three hours a day and working the rest of their time out of doors), really made the greatest intellectual progress during the year. Business men have often accomplished wonders during the busiest lives by simply devoting one, two, three, or four hours daily to study or other literary work.\\n[0.8121] Refresh, renew, rejuvenate yourself by play and pleasant recreation. Play as hard as you work; have a jolly good time, and then you will get that refreshing, invigorating sleep which gives an overplus of energy, a buoyancy of spirit which will make you eager to plunge into the next day's work.\\n[0.8110] A successful manufacturer says: \"If you make a good pin, you will earn more money than if you make a bad steam engine.\" \"If a man can write a better book, preach a better sermon, or make a better mousetrap than his neighbor,\" says Emerson, \"though he build his house in the woods, the world will make a path to his door.\"\\n[0.8103] No matter where you go, study the situation. Think why the man does not do better if he is not doing well, why he remains in mediocrity all his life. If he is making a remarkable success, try to find out why. Keep your eyes open, your ears open. Make deductions from what you see and hear. Trace difficulties; look up evidences of success or failure everywhere. It will be one of the greatest factors in your own success.\\n\\nCHAPTER XXX\\n\\nSELF\\n[0.8088] There is a sense of great power in a vocation after a man has reached the point of efficiency in it, the point of productiveness, the point where his skill begins to tell and brings in returns. Up to this point of efficiency, while he is learning his trade, the time seems to have been almost thrown away. But he has been storing up a vast reserve of knowledge of detail, laying foundations, forming his acquaintances, gaining his reputation for truthfulness, trustworthiness, and integrity, and in  \n",
              "1  [0.8307] Horace Greeley said that the best product of labor is the high-minded workman with an enthusiasm for his work.\\n\\n\"The best method is obtained by earnestness,\" said Salvini. \"If you can impress people with the conviction that you feel what you say, they will pardon many shortcomings. And above all, study, study, study! All the genius in the world will not help you along with any art, unless you become a hard student. It has taken me years to master a single part.\"\\n[0.8152] Adopt this motto as yours. Hang it up in your bedroom, in your office or place of business, put it into your pocket-book, weave it into the texture of everything you do, and your life-work will be what every one's should be--A MASTERPIECE.\\n\\nCHAPTER XXIII\\n\\nTHE REWARD OF PERSISTENCE\\n\\nEvery noble work is at first impossible.--CARLYLE.\\n[0.8135] \"Many persons seeing me so much engaged in active life,\" said Edward Bulwer Lytton, \"and as much about the world as if I had never been a student, have said to me, 'When do you get time to write all your books? How on earth do you contrive to do so much work?' I shall surprise you by the answer I made. The answer is this--'I contrive to do so much by never doing too much at a time. A man to get through work well must not overwork himself; or, if he do too much to-day, the reaction of fatigue\\n[0.8127] \"You are capable of something much better than what you are doing. You must start out to-day with a firm resolution to make the returns from your work greater to-night than ever before. You must make this a red-letter day. Bestir yourself; get the cobwebs out of your head; brush off the brain ash. Think, think, think to some purpose! Do not mull and mope like this. You are only half-alive, man; get a move on you!\"\\n[0.8126] The way to get the best out of yourself is to put things right up to yourself, handle yourself without gloves, and talk to yourself as you would to a son of yours who has great ability but who is not using half of it.\\n\\nWhen you go into an undertaking just say to yourself, \"Now, this thing is right up to me. I've got to make good, to show the man in me or the coward. There is no backing out.\"\\n[0.8125] Edwin Chadwick, in his report to the British Parliament, stated that children, working on half time (that is, studying three hours a day and working the rest of their time out of doors), really made the greatest intellectual progress during the year. Business men have often accomplished wonders during the busiest lives by simply devoting one, two, three, or four hours daily to study or other literary work.\\n[0.8122] Refresh, renew, rejuvenate yourself by play and pleasant recreation. Play as hard as you work; have a jolly good time, and then you will get that refreshing, invigorating sleep which gives an overplus of energy, a buoyancy of spirit which will make you eager to plunge into the next day's work.\\n[0.8105] No matter where you go, study the situation. Think why the man does not do better if he is not doing well, why he remains in mediocrity all his life. If he is making a remarkable success, try to find out why. Keep your eyes open, your ears open. Make deductions from what you see and hear. Trace difficulties; look up evidences of success or failure everywhere. It will be one of the greatest factors in your own success.\\n\\nCHAPTER XXX\\n\\nSELF\\n[0.8103] A successful manufacturer says: \"If you make a good pin, you will earn more money than if you make a bad steam engine.\" \"If a man can write a better book, preach a better sermon, or make a better mousetrap than his neighbor,\" says Emerson, \"though he build his house in the woods, the world will make a path to his door.\"\\n[0.8082] There is a sense of great power in a vocation after a man has reached the point of efficiency in it, the point of productiveness, the point where his skill begins to tell and brings in returns. Up to this point of efficiency, while he is learning his trade, the time seems to have been almost thrown away. But he has been storing up a vast reserve of knowledge of detail, laying foundations, forming his acquaintances, gaining his reputation for truthfulness, trustworthiness, and integrity, and in  \n",
              "2                                                                                                                                   [0.0032] Horace Greeley said that the best product of labor is the high-minded workman with an enthusiasm for his work.\\n\\n\"The best method is obtained by earnestness,\" said Salvini. \"If you can impress people with the conviction that you feel what you say, they will pardon many shortcomings. And above all, study, study, study! All the genius in the world will not help you along with any art, unless you become a hard student. It has taken me years to master a single part.\"\\n[0.0031] \"You are capable of something much better than what you are doing. You must start out to-day with a firm resolution to make the returns from your work greater to-night than ever before. You must make this a red-letter day. Bestir yourself; get the cobwebs out of your head; brush off the brain ash. Think, think, think to some purpose! Do not mull and mope like this. You are only half-alive, man; get a move on you!\"\\n[0.0030] Adopt this motto as yours. Hang it up in your bedroom, in your office or place of business, put it into your pocket-book, weave it into the texture of everything you do, and your life-work will be what every one's should be--A MASTERPIECE.\\n\\nCHAPTER XXIII\\n\\nTHE REWARD OF PERSISTENCE\\n\\nEvery noble work is at first impossible.--CARLYLE.\\n[0.0030] \"Many persons seeing me so much engaged in active life,\" said Edward Bulwer Lytton, \"and as much about the world as if I had never been a student, have said to me, 'When do you get time to write all your books? How on earth do you contrive to do so much work?' I shall surprise you by the answer I made. The answer is this--'I contrive to do so much by never doing too much at a time. A man to get through work well must not overwork himself; or, if he do too much to-day, the reaction of fatigue\\n[0.0030] No matter where you go, study the situation. Think why the man does not do better if he is not doing well, why he remains in mediocrity all his life. If he is making a remarkable success, try to find out why. Keep your eyes open, your ears open. Make deductions from what you see and hear. Trace difficulties; look up evidences of success or failure everywhere. It will be one of the greatest factors in your own success.\\n\\nCHAPTER XXX\\n\\nSELF\\n[0.0030] Edwin Chadwick, in his report to the British Parliament, stated that children, working on half time (that is, studying three hours a day and working the rest of their time out of doors), really made the greatest intellectual progress during the year. Business men have often accomplished wonders during the busiest lives by simply devoting one, two, three, or four hours daily to study or other literary work.\\n[0.0030] The way to get the best out of yourself is to put things right up to yourself, handle yourself without gloves, and talk to yourself as you would to a son of yours who has great ability but who is not using half of it.\\n\\nWhen you go into an undertaking just say to yourself, \"Now, this thing is right up to me. I've got to make good, to show the man in me or the coward. There is no backing out.\"\\n[0.0030] \"Let us not be content to mine the most coal, to make the largest locomotives, to weave the largest quantities of carpets; but, amid the sounds of the pick, the blows of the hammer, the rattle of the looms, and the roar of the machinery, take care that the immortal mechanism of God's own hand--the mind--is still full-trained for the highest and noblest service.\"\\n[0.0029] This striving for excellence will make you grow. It will call out your resources, call out the best thing in you. The constant stretching of the mind over problems which interest you, which are to mean everything to you in the future, will help you expand into a broader, larger, more effective man.\\n[0.0029] A successful manufacturer says: \"If you make a good pin, you will earn more money than if you make a bad steam engine.\" \"If a man can write a better book, preach a better sermon, or make a better mousetrap than his neighbor,\" says Emerson, \"though he build his house in the woods, the world will make a path to his door.\"  "
            ]
          },
          "execution_count": 83,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# To display the results:\n",
        "results_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8F2YgJj2s9Fk"
      },
      "source": [
        "Quantization is a powerful tool for optimizing vector database performance, especially in applications that handle high-dimensional embeddings like semantic search and recommendation systems. This tutorial demonstrated the implementation of scalar and binary quantization methods using Nomic embeddings with MongoDB as the vector database. \n",
        "When leveraged appropriately, effective optimization extends beyond latency improvements. It enables scalability, reduces operational costs, and enhances application user experience. The Benefits of Database Optimization:\n",
        "Latency Reduction for Improved User Experience: Minimizing delays in data retrieval enhances user satisfaction and engagement.\n",
        "Efficient Handling of Large-Scale Data: Optimized databases can more effectively manage vast amounts of data, improving performance and scalability.\n",
        "\n",
        "Cost Reduction and Resource Efficiency: Efficient data storage and retrieval reduce the need for excessive computational resources, leading to cost savings.\n",
        "By examining the trade-offs between retrieval accuracy and performance across different embedding formats (float32, int8, and binary), we showcased how MongoDB's capabilities, such as vector indexing and automatic quantization, can streamline data storage, retrieval, and analysis. \n",
        "\n",
        "From this tutorial, we’ve explored Atlas Vector Search native capabilities for scalar quantization as well as binary quantization with rescoring. Our implementation showed that automatic quantization increases scalability and cost savings by reducing the storage and computational resources for efficient processing of vectors. In most cases, automatic quantization reduces the RAM for mongot by 3.75x for scalar and by 24x for binary; the vector values shrink by 4x and 32x, respectively, but the Hierarchical Navigable Small Worlds graph itself does not shrink.\n",
        "\n",
        "We recommend automatic quantization if you have a large number of full-fidelity vectors, typically over 10M vectors. After quantization, you index reduced representation vectors without compromising the accuracy of your retrieval.\n",
        "To further explore quantization techniques and their applications, refer to resources like [Ingesting Quantized Vectors with Cohere](https://www.mongodb.com/developer/products/atlas/ingesting_quantized_vectors_with_cohere/). An [additional notebook](https://github.com/mongodb-developer/GenAI-Showcase/blob/main/notebooks/advanced_techniques/advanced_evaluation_of_quantized_vectors_using_cohere_mongodb_beir.ipynb) for comparing retrieval accuracy between quantized and non-quantized vectors is also available to deepen your understanding of these methods."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "state": {}
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
